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(57) Abstract 

A hand-held computer input pointing device (100) has at least one motion detector (314), the at least one motion detector being 
capable of detecting motion in at least three dimensions. At least one pressure sensor (316) is capable of sensing pressure quantitatively. 
The input device is operable within a hand of a user to transmit signals from the motion detector and the pressure sensor, without contacting 
a base, work surface or pad. The pointing device may, for example, be used to control a computer (701) executing a program to display a 
movable virtual puppet (710). 
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HUMAN GESTURAL INPUT DEVICE WITH MOTION AND PRESSURE 

FIELD OF THE INVENTION 
The present invention relates the field of input devices for computers. 

DESCRIPTION OF THE RELATED ART 
Computer input pointing devices are well known, A typical computer input pointing 
device contains at least two sensors for sensing motion in at least two directions, such as x and y 
(forward-backward and left-right). The pointing device also has at least one actuator for causing 
the pointing device to transmit a command signal (typically referred to as a "click") to a 
computer." 

The most common pointing device for a desktop computer or workstation is a mouse. 
The mouse may have a ball on its underside. Motion of the ball causes rotation of one or more of 
a plurality of rotation sensors adjacent to the ball. The number of rotations determine the 
magnitude of the motion in the x or y direction. The most common mouse type includes two 
buttons for sending commands known as "left-click" and "right-click. Other types of mice 
include optical sensors that count markings on a specially marked mouse pad to determine an 
amount of movement. 

Another common pointing device used primarily for computer games is a joystick. The 
joystick has a lever that is tilted in a forward-backward or left-right direction, each of which is 
sensed independently. A button is typically provided on the end of the lever for transmitting a 
command signal. Some joysticks also allow rotation of the lever, which is also sensed. 

A variety of alternative pointing devices have been developed. Alternative pointing 
devices are typically used on laptop computers. An early pointing device ilsed on laptop 
computer is the track ball. The track ball functions like an upside-down mouse. Instead of 
moving the device, the ball is rotated directly. 

Many laptop computers include a miniature joystick that is positioned on the home row of 
the keyboard Another common pointing device in laptop computers is a touch pad. The touch 
pad is a rectangular device that is sensitive to touch. Left and right sliding motion of a finger on 
the touch pad is detected. The touch pad also senses when it is struck, and produces a "click" 
signal. 
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Although conventional pointing devices are suitable for locating a point on an x-y grid, 
and transmitting a simple single-valued command signal, conventional pointing devices leave 
much to be desired for controlling application programs that require more complex inputs. 



SUMMARY OF THE INVENTION 

One aspect of the present invention is a hand-held computer input pointing device, 
comprising at least one motion detector. The at least one motion detector is capable of detecting 
motion in at least three dimensions. At least one pressure sensor is capable of sensing pressure 
quantitatively. The input device is operable within a hand of a user to transmit signals from the 
motion detector and the pressure sensor, without contacting a base, work surface or pad. 

Another aspect of the invention is a method for operating a computer having an output 
device, comprising the steps of: receiving a plurality of signals from a computer input pointing 
device, the signals representing quantifiable pressure and motion in at least three dimensions; and 
outputting a video, audio or tactile output signal to the output device, the output signal having at 
least two characteristics that are capable of being varied separately from each other. 

Still another aspect of the invention is a method for operating a computer input pointing 
device, comprising the steps of: moving the pointing device; transmitting at least three motion 
signals from the pointing device to a computer, the motion signals including signals representing 
translations in one or more dimension, or rotations in one or more dimensions, or a combination of 
translations and rotations; squeezing at least one portion of the pointing device; and transmitting 
quantifiable pressure signals from the pointing device to the computer. 

BRIEF DESCRIPTION OF THE DRAWINGS 
FIG. 1 is an isometric view of an exemplary squeezable pointing device according to the 
present invention. 

FIG. 2 is an exploded view of the pointing device of FIG. 1. 

FIG. 3A is an enlarged view of the origami sensor of FIG. 2. 

FIG. 3B is a plan view of the sensor of FIG. 3A, prior to folding. 

FIG. 4A is a plan view of a tetrahedral origami sensor in a flattened state. 

FIG. 4B is an isometric view of the sensor of FIG. 4A, folded. 
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FIG. 5A is a plan view of an alternative cubic origami sensor in a flattened state. 
FIG. 5B is an isometric view of the sensor of FIG. 5A, folded. 
FIG. 6 is an isometric view of a tetrahedral edge sensor. 

FIG. 7 is an elevation view of a computer system including the pointing device of FIG. 1. 
FIG. 8 is a diagram of a range of motion tree for an application program executed on the 
computer shown in FIG. 7. 

FIG. 9 shows the software layers for the application executed on the computer shown in FIG. 

7. 

FIG. 10 is a flow chart of a method for operating the virtual puppet of FIG. 7. 
FIGS. 1 1-13 are block diagrams of the software layer structure shown in FIG. 9. 
FIG. 14 is a flow chart of a method for operating the pointing device of FIG. 2. 

DETAILED DESCRIPTION 

FIG. 1 shows an exemplary computer input pointing device 100 according to the present 
invention. Pointing device 100 is capable of detecting motion and providing a quantifiable measure 
of squeeze pressure. The input device 100 is operable in a hand of a user, without contacting a base, 
work surface, or pad. Thus, the input device 100 is "free floating," in that it may be operated in any 
position without regard to the presence of a base, work surface, or pad. Pointing device 100 has a 
squeezable, deformable elastomeric cover 102. Inside the cover 102, the pointing device 100 has 
an internal motion detector that may measure acceleration in x, y and z planes, or measure pitch, yaw 
and roll. Squeezing the package increases pressure that translates into a pressure signal that is sent 
by the pointing device 100 to the computer (shown in FIG. 7) along with the motion data. 

FIG. 2 is an exploded view of the computer input pointing device 100, with the 
squeezable elastomeric covering 102 partially removed. The squeezable covering 102 may be 
formed from natural or synthetic rubber or a variety of polymeric materials, which may be cut or 
injection molded. Cover 102 has an internal cavity 104 for containing sensors. An optional 
sensor enclosure 106 may be provided inside of cover 102. The sensor enclosure 106 may be 
sealed to protect the sensors from dust and moisture, allowing use of an unsealed cover 102. The 
deformable cover 102, with or without the sensor enclosure 106, allows the sensor to move 
freely, and compresses internal sensor. In one embodiment, the squeezable cover is made from a 
"SQEESH™" ball by the Toysmith Company of Kent, WA USA. "SQEESH™" balls are available 
in a variety of shapes and surface types, including smooth spherical, spherical with craters, and 
star shaped. A variety of shapes may be used to provide users with the grip of their choice or 
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hints as to what object that the device controls. 

Pointing device 100 has at least one motion detector 108, which is capable of detecting 
motion in at least three dimensions. Pointing device 100 also includes at least one pressure 
sensor capable of sensing pressure quantitatively. In the example, the pressure sensing capability 
is also provided by the sensor 108, which is described in greater detail below. The at least one 
motion detector may, for example, include an accelerometer or a piezoelectric device. 

As best seen in FIG. 3A, the exemplary piezoelectric device 108 includes at least three 
piezoelectric strips 301-303 or flaps oriented in three orthogonal directions. More specifically, the 
exemplary device 108 is a single film sensor having three strips 301-303 folded origami style, as 
shown in FIG. 3A, to form a piezo thin film with six sensors. 

FIG. 3B is a view of the device 108 with six sensors 314, 316. The device is formed on a 
substrate 300 and is shown laid flat before the origami folds along dashed lines 312. Optional weight 
W1-W3 may be attached to the end or center of each flap 301-303, respectively. Preferably, each 
of the three piezoelectric strips 301-303 has at least two sections, including a weighted section 314 
and an unweighted section 316; so as to discriminate motion load from pressure load. 

The piezo film substrate 300 is plastic and allows two strips of piezo material to measure 
bend of the substrate by the relative tension between two strips. One piezo sensor 316 is provided 
for pressure discrimination. The pressure on the deformable covering 102 bends the material of the 
piezo pressure sensor 316, producing a measurable voltage. Another piezo sensor 314 for motion 
discrimination has a weight W1-W3 placed at the middle or end of the sensor. If each of the three 
piezoelectric strips 314 has a respective weight; moving the pointing device 100 induces an inertial 
force. The weight W1-W3 moves against the material of the motion sensor 314, causing it to bend 
at a different rate than pressure sensor 316. 

Cuts 318 on the exemplary piezo film substrate 300 allow each sensor flap 314, 316 to 
bend separately. The dotted lines 312 show folds where the flat sensor strip 300 is origami 
folded in 90 degree bends. The folded origami flaps 301-303 are affixed to a support assembly 
320 (shown in FIG. 3A). The support assembly 320 holds the origami flaps 301-303 in place. 
The leads L from thin film sensor are connected to optional internal multiplexing circuitry 1 10 
that connects the sensor circuitry to an external lead 1 12. 

Referring again to FIG. 1, the sensors output data representing four parameters (three 
motion and one pressure). Preferably, a multiplexer 1 10 receives output signals from the motion 
detector and the pressure sensor 108, and outputs a multiplexed signal for transmission to a 
computer (shown in FIG. 7). The output of the multiplexer is connected to a lead 1 12. The lead 
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1 12 extends through a channel 103 in the elastomeric covering 102. The opposite end of the lead 
1 12 has a suitable connector 114 attached to it, which may be a 032-centimeter (1/8 inch) 
miniplug, a Universal Serial Bus (USB) connector, a serial (RS-232) port, an "ETHERNET™" 
port, a MIDI port for connection to a musical instrument, a PDA port, a cellular phone port or 
other suitable connector. 

In a system having a plurality of similar pointing devices 100, a hub capable of accepting 
inputs from multiple input devices may be provided to interface the plurality of pointing devices 
100 to a single computer. For example, if there are multiple pointing devices having USB 
connectors, a USB hub 116 (connected to lead 1 18 and USB connector 120) may be used to 
connect the pointing devices to a single computer. Alternatively, the pointing device may omit 
the multiplexer, and output four analog signals from four respective output terminals. The 
pointing device 100 may have a wireless transmitter for communicating with the computer. 

The lead 1 12 from the sensing package either connects one data line (carrying time or 
frequency multiplexed data), or multiple data lines each carrying a respective sensor's output. 
The connector (such as 1/8" miniplug) 1 14 optionally connects directly to the computer (FIG. 7) 
through a microphone input eliminating the need for a hub. The pointing device 100 may use the 
audio software driver or communicate through the optional hub 1 16 for controlling multiple 
puppets (as described below). The connector 1 14 may connect to a different computer port such 
as a USB or serial port, eliminating the need for a hub. Although FIG. 2 shows a two-input 
version, the system may include any number of inputs. 

Any of the sensor types described herein may be used with clock shifting to the next lead. 
An additional clock cycle may be provided for subsequent processing of an optional unique 
identification number (described below). The sensors provide an analog signal that may be 
modulated with a single carrier frequency, or the signal from each lead may be modulated with a 
respectively different carrier frequency. 

An optional analog to digital converter (ADC) may be provided for each of the analog 
sensor leads. In this case, a clock shift to the next lead sends a digital packet. For example, with 
eight bits per lead, which for 6 sensors yields six bytes per sample cycle. 

An optional identifier (ID) circuit may be provided. The ID circuit may be a register or a 
read only memory (ROM) device, EEPROM, or the like. The unique identifier may be hard 
encoded at the factory (e.g., using a ROM), or the identifier may be downloaded from a computer 
into a flash EEPROM in the pointing device 100. As explained below with reference to FIG. 7, 
by providing a unique identifier for each pointing device unit 100, more than one unit may be 
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used at the same time. The device identifier can function like an encryption key, to protect the 
user's preferences or to securely identify the user. 

The output of the ID circuit and the output from one of the sensors are output to the 
multiplexing circuitry 1 10. The analog input may be time-domain multiplexed with the ID 
waveform. Alternatively, if the analog sensor signals are converted to digital form before 
transmission to the multiplexer, a digital input packet may be combined with ID information to 
form a larger packet including both. The pointing device 100 may contain additional identifier 
circuitry to add a port identifier number separate from the pointing device identifier number. 

The analog waveform may have a separate carrier frequency or may be time domain 
multiplexed or interleaved with each sample, either on a bit, byte, or packet basis. Alternatively, 
if a digital packet format is used, the sensor data and ID information may be added to a 
multiplexing packet to form a larger packet including both types of data. 

The pointing device 100 may obtain power from the computer (FIG. 7), from an internal 
battery, or by drawing power directly from the energy created by the sensor from the 
piezoelectric effect. 

ALTERNATIVE SENSORS 

The piezoelectric device may be an origami sensor having cut coils, and may have either 
a cubic or tetrahedral shape. Preferably, the cut coils on each face of the motion detector are 
attached to a single weight at a center of the motion detector. Optionally, each cut coil on each 
face of the motion detector has its own weight attached. 

FIGS. 5 A and 5B show a sensor system 500 including origami cutouts, which may be 
used in place of the sensor 108 shown in FIG. 2. A thin sensor film 500 having six faces 501- 
506 is folded to form a cube (although a tetrahedron, shown in FIGS. 4A and 4B, or other three- 
dimensional shapes may be used in alternative embodiments). A spiral pattern 511-516 is cut 
into each face of thin sensor film 501-506, respectively. Each coil can be pressed towards the 
center (e.g., CI, C3 and C6) and attached to a weight that is suspended at the center of the folded 
shape. 

When the pointing device is moved, the weight applies forces to each of the coils eliciting 
a voltage. For the cube 500, each face 501-506 has a corresponding opposite face measuring 
force with an inverse effect on voltage. 

For the tetrahedron shown in FIGS. 4A and 4B, each side faces three other sides, 
measuring an inverse effect on their faces. 
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Additional circuitry (not shown) may be included in the exemplary sensor configuration. 
An analog sample and hold is provided for each of the analog sensor leads. For the exemplary 
origami sensor of FIGS. 2, 3A and 3B, the leads are: 

1. A free - for pressure discrimination 

2. A weighted - for motion discrimination 

3. B free - pressure 

4. B weighted - motion 

5. Cfree -pressure 

6. C weighted - motion 

For a tetrahedral sensor (FIGS. 4A and 4B), the leads are: 

1. A tension 

2. B tension 

3. C tension 

4. D tension 

For a cubic sensor (FIGS. 5 A and 5B), the leads are: 

1. AO tension 

2. Al tension 

3. BO tension 

4. B2 tension 

5. CO tension 

6. l Cl tension 

FIG. 6 shows a tetrahedral edge sensor 600. Sensor 600 has six edges to detect pressure 
directly from a squeeze of the defonnable elastomeric material of cover 102. Optionally, sensor 
600 may have weighted edges 602 to detect motion through inertial force whenever the pointing 
device 100 is moved. The weights 610 increase the momentum of the sensors 602, so that the 
deformation of the sensors is exaggerated. Alternatively, weights may be placed at each node 
608. 

Sensor 600 is used in conjunction with an application driver that can discriminate 
pressure from motion. The driver can discriminate pitch, yaw, and roll. A plurality of signal 
leads 604 are provided, one for each edge of the tetrahedron 600. The outputs 604 of the sensor 
either connect to multiplexing/conversion circuitry 1 10 or lead out to separate analog lines. 
There is an input lead from each edge of tetrahedron. Optionally each connecting node 608 
broadcasts a wireless signal of its state without requiring any lead wire. As shown in FIG. 6 by 
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dotted lines 606, non-sensing leads connect each node 608 to another edge 602. The non-sensing 
leads 606 either follow a sensing edge 602 to another node 608 or may run directly to another 
node 608. The connecting nodes 608 for leads 606 and sensing edges 602 may simply be 
vertices for an origami tetrahedron or connecting nodes for self supporting edge sensors. 

In one embodiment, a sensor similar to that used in the "GYROPOINT™" mouse may be 

used. 

In another embodiment, a sensor such as the "ACH-04-08-05" accelerometer/shock sensor 
from Measurement Specialties, Inc. of Valley Forge PA may be used. The ACH-04-08-05 sensor 
has three piezoelectric sensing elements oriented to measure acceleration in x, y and z linear axes, 
and uses very low power. Alternatively, an "ACH-04-08-0r sensor by Measurement Specialties, 
Inc. may be used to measure linear acceleration in the x and y directions and rotation about the z 
axis. 

In addition to the motion detection capability, pointing device 100 also includes a 
pressure sensing capability. An internal pressure sensor is included, which may be, for example, 
a thin film piezo sensor that detects a bending of a strip or a sensor such as the "MTC 
Express™" from Tactex Controls Inc. of Cerritos, CA. This control surface is pressure sensitive, 
and also senses multiple points of contact, using a fiber optic-based, pressure sensitive, Smart 
Fabric called "KINOTEX™." This fabric is constructed of cellular urethane or silicone 
sandwiched between protective membranes. The fabric has an embedded fiber optic array that 
generates an optical signal when the material is touched, or when force is applied. The 
coordinates and area of the pressure, and its magnitude can be determined from the received 
signals. The material can generate a pressure profile of the shape of a hand touching it. 

As noted above, the pointing device 100 may include a circuit that outputs a unique identifier 
for each pointing device. There are many applications for this feature. For example, the feature 
allows the pointing device 100 to serve a user identification and/or authentication function. A user 
could carry around his or her pointing device with a unique identifier output signal, and connect it 
to any desired computer having an Internet connection. The application program could request a 
remote server to download previously established software preferences or upload the user's current 
software preferences. This enables a person to log on at any computer connected to the Internet, 
access his or her software (e.g., virtual puppet and stage application), or download web bookmarks. 
Alternatively, the user could use the pointing device 100 as a key to access copy protected software, 
functioning like a parallel port dongle. Thus, the use of a pointing device 100 with a built-in unique 
identifier enhances the user's mobility and access to services. 
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Further, the pointing device may be used for authentication at any computer to which it is 
connected. For example, because the unit uniquely identifies itself, the unit can be used in a way that 
allows it to bind to each individual user. For example, the unique identifier may be used in 
combination with a password, IP address or a user signature. For example, the user may sign his or 
her name using the pointing device to control movement of a virtual writing implement. The user's 
handwriting may be recognized using a conventional handwriting recognition algorithm, such as any 
of those described in U.S. Patents 5768423, 5757959, 5710916, 5649023 or 5553284, all of which 
are expressly incorporated by reference herein in their entireties. The combination of the unique 
device identifier and the user signature provides reliable authentication. 

Further still, the unique identifier may be used to make the user's preferences more portable. 
For example, the identifier may be associated with the design (i.e., shape, color, and the like) of the 
cover 102. The identifier for the pointing device may also be associated with different shapes and 
colors of cursors. The cursor that appears on the screen may have the same shape or color as the 
cover 102 of the input device 100, which is associated with a known identifier. For example, a blue 
spherical pointing device may be associated with a round, blue cursor, a puppet shaped pointing 
device may be associated with a puppet shaped cursor, an alien-shaped pointing device may be 
associated with an alien-shaped cursor, and so on. 

Alternatively, the input device packaging may be completely different from the shape and 
color of the computer cursor, or puppet object. If the puppet application program is stored in a server 
that is accessible via the Internet, or other local area network (LAN) or wide area network (WAN) 
or wireless services such as CDMA, the user may select from any of a plurality of different puppets 
from a gallery. This is one of the user preferences that may follow her around from client computer 
to client computer. 

VIRTUAL PUPPET SYSTEM 
FIG. 7 shows a computer system 700 according to another aspect of the invention. The 
computer system 700 includes a computer 701 having a processor 702, a memory 706, a display 708, 
and an input port 712. 

At least one computer input pointing device 100, as described above with reference to FIG. 
2, is coupled to the input port 712 of the computer 701. The pointing device 100 has at least one 
motion detector 108, capable of detecting motion in at least three dimensions, and at least one 
pressure sensor capable of sensing pressure quantitatively. A multiplexer 1 10 receives output signals 
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from the motion detector 108 and the pressure sensor, and outputs a signal for transmission to the 
computer 701 . Alternatively, the lead from each sensor may output an analog signal. 

The computer 701 further comprises a storage device 704, which may be a hard disk drive, 
a high capacity removable disk drive (e.g., "ZIP" drive by Iomega Corporation), a read only memory 
(ROM), CD-ROM drive, a digital versatile disk (DVD) ROM drive, or the like. Either the memory 
706 or the storage device 704 has stored therein computer program code for causing the computer 
to show an object 710 on the display 708. The object has a plurality of portions 710a-710d that are 
selectively movable in response to motion or squeezing of the pointing device 100. The exemplary 
object is an animated character 710 controlled by input parameters from the pointing device. Such 
an animated character is referred to herein as a "virtual puppet". The virtual puppet has a plurality 
of independently movable body parts 710a-710d. Although the example only shows movable arms 
and legs, any part of the virtual puppet 710 may be movable. 

According to another aspect of the invention, a method is provided for controlling a 
computer. The method includes the steps of: receiving a plurality of signals from a computer input 
pointing device 100, the signals representing quantifiable pressure and motion in at least three 
dimensions; and outputting a video, audio or tactile output signal to the output device 100, the output 
signal having at least two characteristics that are capable of being varied separately from each other. 

FIG. 10 shows an exemplary method for operating the computer 701. The method 
includes the steps of: 

• receiving a plurality of signals from a computer input pointing device 100, the signals 
representing quantifiable pressure and motion in at least three dimensions; 

• displaying a virtual puppet 710 on the display 708, the virtual puppet having at least two 
portions 710a-710d that are capable of being moved separately from each other; and 

• showing movement of the portions 7 10a-7 lOd of the virtual puppet 7 10 in response to the 
pressure and motion signals. 

At step 1010, the animated character 710, figure or other object is displayed on the 
display 708. At step 1020, when the user squeezes pointing device 100, stej>s 1030-1050 are 
executed. At step 1030, the driver module 904 updates the idealized pressure value Pi and 
translates Pi into the normalized pressure value P. At step 1040, the application program 
processes the normalized pressure P into an animated figure action. At step 1050, the application 
updates the animated figure according to the action, and displays the result on the display 708, 

At step 1060, when the user moves the pointing device 100, steps 1070-1080 are 
executed. At step 1070, the driver module 904 of the software updates the idealized motion 
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vector Mxi, Myi, Mzi and translates the vector into a normalized motion vector Mx, My, Mz. At 
step 1080, the application program processes the motion vector Mx, My, Mz into animated figure 
motion, and outputs the moving figure to the display 708. 

An exemplary method for operating the pointing device 100 comprises the steps of: 

1 . moving the pointing device 100; 

2. transmitting at least three motion signals from the pointing device 100 to a computer 
701, the motion signals including signals representing translations in one or more 
dimension, or rotations in one or more dimensions, or a combination of translations 
and rotations; 

3. squeezing at least one portion of the pointing device 100; and 

4. transmitting quantifiable pressure signals from the pointing device 100 to the 
computer 701. 

Preferably, all parameters measured by the sensors 108 in the pointing device 100 are 
mapped to motions of the virtual puppet 710, such as mouth moving open and closed, the motions 
of limbs, or the expression of emotional intensity. The pressure measurement is particularly suitable 
for use in providing quantitative inputs to the computer to create results having continuously variable 
intensity. Thus, a light squeeze may result in a small kick, whereas a hard squeeze results in a high 
kick. 

The example of FIG. 7 includes a hub 1 16 and at least a second computer input pointing 
device 100 capable of sensing motion in at least three dimensions and a pressure sensor. Both the 
first and second computer input pointing devices 100 are connected to the hub 1 16, which in turn 
is connected the input port of the computer 701. Each of the first and second computer input 
pointing devices 100 controls a respective virtual puppet 710 and 71 1. 

As shown in FIG. 8, the computer program code includes a range of motion tree 800 for 
the virtual puppet 710. The range of motion tree 800 restricts a range of possible motion of the 
virtual puppet 710 about a current position of the virtual puppet. Preferably, the computer 
program code includes a respective unique range of motion tree for each of a plurality of users. 

The exemplary range of motion tree 800 covers head, left and right shoulder, left and 
right elbow, left and right wrist, and left and right grip. The tree limits the range of motion from 
each node. For example, the nodes may be delimited as follows: 

1. Head Hxyz 

2. Shoulder Sxyz 
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5. Grip 



3. Elbow 



4. Wrist 



Exyz 

Wxyz 

Gxyz 



The limits may be expressed in rectangular coordinates, such as (xO,yO,z0), (xl,yl,zl). 
Alternatively, the limits may be expressed as delta ranges. For some objects, it may be more 
convenient to express the limits in polar coordinates, which have a 1:1 mapping to rectangular 
coordinates. 

The range of motion tree 800 can express a hierarchy of motions. For example, in the 
case of the head, the range of motion may be express as: 



One of ordinary skill can readily see that similar ranges of motion can be specified for 
elbow, wrist, and grip 

All motion then extends the range of the tree 



State information restricts the range of possible motion about the last position of puppet. 
The state information is mapped to a user model, and further restricts possible ranges of motion. 
For example, 

Hxyz:H* => (HxOO,Hxll), (HyOO Hyll), (HzOOJkll) 

A user model adds probabilities for motion at each node of the tree. For example, the 
user model may identify whether the user moves her shoulders or shakes her wrists. In an 
exemplary user model for "User n'\ the probabilities may be expressed as: 

Un: (xO,xO), (yO,yl), (zO,zl) 

As an alternative to the tree search, the probabilities of motion may be mapped to a voxel 
space. The probabilities could be precompiled with user probabilities, where three-dimensional 
motion space is broken up into small cubes, analogous to three-dimensional pixels. Each voxel 
point can be ascribed a probability, and each voxel can contains a range of motion probability. 
Voxels may also include application restrictions on the range of motion probability. 

Another task for the software is identifying the orientation of the pointing device, to allow 
use of the orientation as a parameter. In one embodiment, conformation of curves is used as a 
process of matching the signal curve over time from the three-dimensional sensor space to three- 



(HxO, Hxl), (HyO, Hyl), (HzO,Hzl) => H* 
where H* denotes range of motion 



For the shoulder , the range of motion may be express as: 
(SxO, Sxl), (SyO, Syl), (SzO,Szl) => S* 



H*.S*.E*.W*.G* 
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dimensional application space. This is analogous to a three-dimensional jigsaw puzzle or 
Chinese wood block puzzle, in which a piece may be rotated in any of three dimensions. Using 
voxel space, it is possible to constrain the orientation of the sensor motion curve to only possible 
ranges from the "range of motion tree" 800 to only allow possible ranges from the application 
space. It is further possible to find a most likely orientation of sensor motion curve based on 
probabilities of each motion represented as likelihood coefficients for each voxel point. 

Orientation can be determined by software. The exemplary software uses conformation 
mapping of curves of X, Y, Z (and optionally P) to the range of motions of either the left or right 
hand. Mapping could emanate from an explicit model or from trainable classifiers, such as 
neural nets. An application may request the user to orient the pointing device 100 by holding the 
device in his or her left or right hand and selecting items on the display 708. 

The user either controls an object on the display 708, or navigates in space, where the 
user acts as the object, with a point of view that shifts. In either case, the range of object motion 
may be less than the range of motion of the pointing device 100. Thus, the software maps the 
actual gesture of the input device to the best fitting object motion. 

A simple example is controlling a cartoon character that merely has a range of motion of 
stage-left and stage-right. Here, the range of motion of the puppet is constrained to two 
dimensions. There are several phases of control that should be considered to gauge its affect in 
the virtual world. For example: whether the pointing device is operated as a left-handed device 
or a right-handed device; whether the line 1 12 is sticking up or down; the current model or user 
range of motions; the current state of pressure; and the current desired range of controlled object 
motion. Controlled objects on the display 708 generally have a limited destination range. 

As an alternative to the range of motion tree 800, one can require manual orientation of 
the pointing device 100. For example, the application can show a picture of the pointing device 
100 with the lead 112 pointing down to the floor. One hemisphere of the squeezable, deformable 
cover may have special markings that faces away from the display 708. This technique 
eliminates the need for conformation of curves, since the orientation is 1:1, puppet to application. 

FIG. 9 is a block diagram showing the virtual puppet modules and their connections to 
each other and the underlying software layer. The user module 900 contains the states of: 

1. grasp - The user is picking up device. 

2. orient - The user is matching device orientation to application and screen feedback 

3. Squeeze - The user is pressing in on squishy covering 

4. Move - The user is moving device 
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5. Release - The user is putting down the device 

6. state - The module reports user state to other modules 

A sensor module 902 has processes A, B, C, D corresponding to each sensor output, 
whether analog or digital. 

A driver module 904 translates sensor values to application values. The driver module 
904 may use a trainable classifier, such as a neural network, where each sensor output has 
correlating inputs to the drivers idealized X, Y, Z, P. The driver module 904 correlates user 
states to range of sensor input. The states, "grasp," "orient," "squeeze," "move," and "release" 
each have a characteristic effect on each of the sensors. 

For instance, "grasp" may have a pressure curve, as the user grabs and picks up the 
puppet. As another example, "squeeze" may have a longer continuous pressure curve when 
compared to grasp. Each instance of state may have separate set of coefficients for a trainable 
classifier. 

The software correlates the application state to the domain of sensor output, to reduce the 
set of possible outcomes. For instance, a two-dimensional stage for cartoon characters may be 
limited to "stage left" and "stage right". The program updates the user module state from the 
application state and application position. 

The application module 906 accepts X, Y, Z, and P data from the driver module 904, and 
reports the application X, Y, Z, P state to the driver module as state information. Optionally, the 
application module 906 reports user state information to pass from the driver module 904 to the 
user module 900. 

A mouse module 908 is a special program for emulating a mouse. The mouse module 
908 uses a special application module for the report state. Mouse module 908 translates X, Y 
movement of the pointing device 100 to mouse X,Y motion, and translates Z and P parameters of 
the pointing device 100 to mouse Left, Right, and double click. 

FIGS. 11-13 show the transformations that are performed by various exemplary software 
modules, depending on the specific types of sensors 902 included in the pointing device 100. 

FIG. 1 1 covers the example described above, in which the motion detector measures x, y, and 
z accelerations, and pressure is also measured. The sensors 902 include motion sensors 1 100 and 
pressure sensor 1110. A program Sm 1 120 translates the raw output of sensor 1 100 to an idealized 
motion vector, Mxi, Myi, Mzi, 1 140 that is provided to the driver 904. A second translator Tm 1 160 
translates the idealized motion to an application motion vector Mx, My, Mz 1 1 80, which is provided 
to the application 906. A program Sp 1 130 translates the raw output of sensor 1 110 to an idealized 
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pressure Pi, 1150 that is provided to the driver 904. A second translator Tp 1170 translates the 
idealized pressure Pi to an application pressure P 1 190, which is provided to the application 906. 

FIG. 12 covers a second example, in which an origami detector measures x, y, and z 
accelerations, and pressure. The sensors 902' include unweighted sensors 1200 and weighted sensors 
1210. A program Sm 1220 discriminates between unweighted and weighted sensors to produce an 
idealized motion vector, Mxi, Myi, Mzi, 1240 that is provided to the driver 904'. A second 
translator Tm 1260 translates the idealized motion to an application motion vector Mx, My, Mz 
1280, which is provided to the application 906\ A program Sp 1230 discriminates between 
unweighted and weighted sensors to produce an idealized pressure Pi, 1250 that is provided to the 
driver 904'. A second translator Tp 1270 translates the idealized pressure Pi to an application 
pressure P 1290, which is provided to the application 906'. 

FIG. 13 covers a third example, in which a tetrahedral detector measures x, y, and z 
accelerations; x, y and z rotations; and pressure. The sensors 902" include tetrahedral sensor 1300. 
A program Sm 1310 translates the output of sensor 1300 to an idealized motion vector, Mxi, Myi, 
Mzi, 1340 that is provided to the driver 904". A second translator Tm 1370 translates the idealized 
motion to an application motion vector Mx, My, Mz 1371, which is provided to the application 
906". A third translator Sr 1320 translates the sensor information to an idealized rotation vector Rxi, 
Ryi, Rzi. A fourth translator Tr 1380 translates the idealized rotation Rxi, Ryi, Rzi to an application 
rotation Rx, Ry, Rz 1350, which is provided to the application 906". A fifth translator Sp 1330 
translates the output of sensor 1300 to an idealized pressure Pi, 1360 that is provided to the driver 
904". A sixth translator Tp 1390 translates the idealized pressure Pi to an application pressure P 
1391, which is provided to the application 906" . 

FIG. 14 is a flow chart diagram of an exemplary method for operating the pointing device 

100. 

At step 1419, the system waits for the user to take an action, i.e., a motion or a squeeze. The 
system returns to the wait state 1419 after each action, and can transition from the wait state 1419 
to any of the states, "grasp" 1420, "orient" 1432, "squeeze" 1437, "move" 1442 or "release" 1447. 

At step 1420, the user grasps the device. At step 1421 the sensor 902 registers the motion. 
At step 1422, the driver module 904 wakes up in "grasp mode". 

At step 1423, if this is the first session for the pointing device 100, then step 1424 is 
executed, and the device is calibrated. Control is transferred to step 1425. At step 1425, if this is 
the first time the user is using the pointing device 100, then at step 1426, the user registers himself 
or hereelf. An association is thus formed between the user and the particular pointing device, making 
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use of the (optional) unique identifier chip in the pointing device 100. Control is transferred to step 
1427. At step 1427, if a login is required, then step 1428 is executed. At step 1428, the user logs 
in, either by using the keyboard, or by a unique motion with the pointing device 100 (such as writing 
the user's signature). Control is transferred to step 1429. 

At step 1429, the system checks whether there are user-specific coefficients available on the 
system for translating this specific user's style of motion, rotation and pressure from raw sensor data 
to idealized measurement data. If data are available, then step 1430 is executed. At step 1430, the 
default coefficients are replaced by the previously determined user-specific coefficients. Control is 
transferred to step 1431. At step 1431, the user login is translated to rotation and orientation 
coefficients. The system returns to the wait state 1419 

At step 1432, the user orients the device. At step 1433, a rotation calibration is initiated by 
asking the user to orient the device in one or more predetermined positions. At step 1434, a stream 
of data points are recorded. At step 1435, conforming X, Y and Z rotations are constructed. At step 
1436, a deteiroination is made whether sufficient data have been collected to conform the rotation 
measurements to the actual rotations. If not, then control is returned to step 1433. Otherwise, the 
system returns to the wait state 1419. 

At step 1437, the user squeezes the pointing device 100. At step 1438, the driver module 904 
updates the idealized pressure Pi, and translates the same to the normalized pressure P. At step 1439, 
the pressure is provided to the application, which processes the pressure. At step 1440, if the 
application is finished with processing the squeeze operation, then control returns to step 1419 (the 
wait state). If not, then step 1441 is executed, to determine whether driver module 904 detects a lull 
or pause in the squeezing. If there is a pause, then control is transferred to step 1419 (wait state). 
Otherwise, step 1438 is executed again. 

At step 1442, the user moves the pointing device 100. At step 1443, the driver module 
904 updates the idealized motion vector Mxi, Myi, Mzi, and translates the same into the 
normalized motion vector Mx, My, Mz. At step 1444, the application program processes the 
normalized motion vector Mx, My, Mz. At step 1445, if the application is done with processing 
the movement of the pointing device 100, then control transfers to step 1419 (wait state). 
Otherwise, step 1446 is executed, to determine whether the driver detects a lull in the motion. If 
there is a pause, then control is transferred to step 1419 (wait state). Otherwise, step 1443 is 
executed again. 

At step 1447, the user releases the pointing device 100. At step 1448, the driver module 
904 enters a sleep state. At step 1449, the application is notified that the pointing device has 

-16- 



WO 00/63874 



PCT/US00/10579 



entered the sleep state. 

Listed below is a set of pseudo-code that may be used to construct a software system 
suitable for handling input signals from the exemplary pointing device 100. 

Definition of Terms 

A, B, C, D, E, F, G sensor strip oriented along a sensing plane 

Au, Bu,... unweighted portion of sensor 

Aw, Bw,... weighted portion of sensor 

Mxi, Myi, Mzi idealized (un-normalized) X, Y, Z value 

Mx, My, Mz, Mp normalized X, Y, Z values used by application 

Ox, Oy, Oz orientation vector to translate idealized to normalized X,Y,Z values 

Pi idealized pressure value 

P normalized pressure value 

Sm translate sensor outputs to idealized motion 

Sr translate sensor outputs to idealized rotational motion 

Sp translate sensor outputs to idealize pressure 

Tm translate idealized motion to application motion 
Tp translate idealized pressure to application pressure 
Tr translate idealized rotational motion 
to application rotational motion 

Accelerometer ABC with pressure D 

Sm: A maps to Mxi 
B maps to Mzi 
C maps to Mzi 

Sp: D maps to Pi 

Tm; (Mxi, Myi, Mzi) rotationally maps to (Mx, My, Mz) 
Tp: Pi maps to P 
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Origami sensor with ABC flaps with each flap having 
an un -weighted (u) 
and weighted (w) section 

Sp: (Au, Bu, Cu) maps to (Pi) 

Sm: (Aw, Bw, Cw, Pi) maps to (Mxi, Myi, Mzi) 

Tm: (Mxi, Myi, Mzi) rotationally maps to (Mx, My, Mz) 

Tp: Pi maps to P 

Tetrahedral sensor ABCDEF with six strips 
Sp: (ABCDEF) maps to Pi 

Sm: (ABCDEF, Rxi, Ryi, Rzi) maps to Mxi 
(ABCDEF, Rxi, Ryi, Rzi) maps to Myi 
(ABCDEF, Rxi, Ryi, Rzi) maps to Mzi 

Sr: (ABCDEF, Mxi, Myi, Mzi) maps to Rxi 
(ABCDEF, Mxi, Myi, Mzi) maps to Ryi 
(ABCDEF, Mxi, Myi, Mzi) maps to Rzi 

Tm: (Mxi, Myi, Mzi) rotationally maps to (Mx, My, Mz) 
Tr: (Rxi, Ryi, Rzi) rotationally maps to (Rx, Ry, Rz) 
Tp: Pi maps to P 

Calibration 

ask user to center device 
user moves device and 

wait for activity and the lull or press 



ask user to trace shape on XY axis 
user moves device 
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while software collects ABCD points 
translate ABCD data points to Ox, Oy sensor coefficients 

ask user to move in and out on Z axis 
user moves device in and out 

while software collects ABCD points 
translate ABCD points to Oz sensor coefficients 

Registration 

after calibration 

ask user to create login signature 
determine cultural orientation 

left-right vs. right-left, etc. 
accept user's input 

while software collects ABCD points 
translate input to Ox, Oy, Oz over time 

Login 

ask user to reproduce login signature 
set orientation based upon cultural 

left-right vs. right-left context 

replaces manual orientation 

to draw shape on X-Y axis 

Orientation 

if manual orientation 

ask user to move input device to center of area 
with cord facing direction (floor) 
and front face facing direction 
set Ox, Oy, Oz to (0,0,0) 

else 

while waiting for input device to stop 
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or pressure state to signal squeeze state 
save last few Mxi, Myi, Mzi states 



Although the exemplary application program described above is a virtual puppet program, 
there are many uses for a pointing device 100 according to the present invention. For example, the 
output of the pointing device may be mapped to the parameters of a visual synthesizer, such as the 
shifting of a color palette, the drawing or motion of an object. Alternatively, the output of device 
100 may be mapped to parameters of a sound synthesizer, such as pitch, timbre, amplitude, or 
placement in three-dimensional sonic space. 

Although the exemplary virtual puppet application is configured so that one pointing device 
100 controls a single virtual puppet 710, the software can be configured to operate more than one 
virtual puppet (or other object) using a single input pointing device 100. For example, assuming that 
the pointing device includes sensors for measuring x, y, and z accelerations and pressure P, the first 
puppet 710 may be controlled by the x and y accelerations, and the second puppet 711 may be 
controlled by z and P. The first and second objects need not be identical objects, or even objects of 
the same type. 

For example, the first object may be a virtual puppet, and the second object may be the 
background scenery or sound. By x and y movements, the puppet 710 is controlled, and by z and 
P, the user can vary the background from day to night, or from silence to the sound of thunder. 

To use two input devices 100, software applications can us the Mouse API for selecting or 
navigating or use a Driver API. For example, the driver for Digitizer (or Graphics) tablets may allow 
more than one pen to be used at the same time. Emulating a mouse can be accomplished by installing 
a driver and going beyond a mouse can be accomplished by modifying the application program. One 
can also overlay other input devices, such as a graphics tablet, with yet another driver. 

Further, the output signals from the exemplary pointing device 100 may be mapped to those 
of conventional input devices in an emulation mode. For example, the outputs may be mapped to 
the x and y positions of a conventional mouse and the left or right click signals from depressing the 
mouse buttons. Mapping the x, y and z (or x, y and P) parameters into x and y parameters only may 
require the computer to determine the orientation of the pointing device 100, such as moving on a 
predominantly x-y plane, a y-z plane, or the z-x plane, or combinations of all three. Alternatively, 
individual user profiles may be used to determine the current orientation. 

Although the exemplary use for two pointing devices is to control two different virtual 
puppets, a second pointing device may be used to expand the number of parameters for an 

-20- 



WO 00/63874 PCT/US00/10579 

application. For example, in an audiovisual application, one pointing device 100 may control 
various aspects of the video signal, while another pointing device may be used to control the audio 
signals. 

Although the exemplary pointing device is sensitive to motion and pressure, additional 
sensors may also be included. For example, a microphone may be incorporated into the pointing 
device, for operation of voice activated software, or for storage of sounds. 

Although the exemplary embodiment described above has the virtual puppet application 
program running locally in the computer 701, the application program may be executed in other 
processors located remotely, either as a sole process or in parallel with other instances of the 
application program on other processors. The local computer can communicate with other 
computers via a telecommunication network, which may be a LAN, WAN, Internet, or wireless 
protocols. 

The present invention may be embodied in the fonm of computer-implemented processes and 
apparatus for practicing those processes. The present invention may also be embodied in the form 
of computer program code embodied in tangible media, such as floppy diskettes, read only memories 
(ROMs), CD-ROMs, hard drives, "ZIP™* drives, or any other computer-readable storage medium, 
wherein, when the computer program code is loaded into and executed by a computer, the computer 
becomes an apparatus for practicing the invention. The present invention may also be embodied in 
the form of computer program code, for example, whether stored in a storage medium, loaded into 
and/or executed by a computer, or transmitted over some transmission medium, such as over the 
electrical wiring or cabling, through fiber optics, or via electromagnetic radiation, wherein, when the 
computer program code is loaded into and executed by a computer, the computer becomes an 
apparatus for practicing the invention. When implemented on a general-purpose processor, the 
computer program code segments configure the processor to create specific logic circuits. 

Although the invention has been described in terms of exemplary embodiments, it is not 
limited thereto. Rather, the appended claim should be construed broadly, to include other variants 
and embodiments of the invention that may be made by those skilled in the art without departing 
from the scope and range of equivalents of the invention. 
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1 1 . A hand-held computer input pointing device, comprising: 

2 at least one motion detector, the at least one motion detector being capable of detecting 

3 motion in at least three dimensions; and 

4 at least one pressure sensor capable of sensing pressure quantitatively, 

5 wherein the input device is operable within a hand of a user to transmit signals from the 

6 motion detector and the pressure sensor, without contacting a base, work surface or pad. 

1 2. The pointing device of claim 1, wherein the at least one motion detector includes at least one 

2 of the group consisting of an accelerometer and a piezoelectric device. 

1 3. The pointing device of claim 2, wherein the at least one motion detector includes at least 

2 three piezoelectric strips oriented in three orthogonal directions. 

1 4. The pointing device of claim 3, wherein each of the three piezoelectric strips has a respective 

2 weight thereon to translate motion to a voltage. 

1 5. The pointing device of claim 3, wherein the piezoelectric device is an origami sensor having 

2 cut coils, the origami sensor having either a cubic or tetrahedral shape. 

1 6. . The pointing device of claim 1, wherein the motion sensor includes a tetrahedral shaped 

2 device having six edge sensors. 

1 7. The pointing device of claim 1, wherein the motion detector and pressure sensor are 

2 contained within a squeezable housing. 

1 8. The pointing device of claim 7, wherein the housing is an elastomeric ball. 

1 9. The pointing device of claim 1 , wherein the motion sensor is capable of sensing pitch, yaw 

2 and roll. 

1 10, The pointing device of claim 1, wherein the motion sensor is capable of sensing x- 

2 acceleration, y-acceleration and z-acceleratiori. 
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1 11. The pointing device of claim 1, further comprising a memory device that stores and outputs 

2 an identifier that is unique to the computer input pointing device. 

1 12. A computer system comprising: 

2 a computer having a processor, a memory, an output device, and an input port; and 

3 a hand-held computer input pointing device coupled to the input port of the computer, the 

4 pointing device comprising: 

5 at least one motion detector, the at least one motion detector being capable 

6 of detecting motion in at least three dimensions; and 

7 at least one pressure sensor capable of sensing pressure quantitatively, 

8 wherein the input device is operable within a hand of a user to transmit 

9 signals from the motion detector and the pressure sensor, without contacting a base, 
1 o work surface or pad. 

1 13. The computer system of claim 12, wherein: 

2 the output device is a display; 

3 the computer further comprises a storage device, and 

4 one of the group consisting of the memory and the storage device has stored therein computer 

5 program code for causing the computer to show an object on the display, the object having a plurality 

6 of portions that are selectively movable in response to motion or squeezing of the pointing device. 

1 14. The computer system of claim 13, wherein the object is an animated character controlled by 

2 input parameters. 

1 15. The computer system of claim 14, further comprising a hub and at least a second computer 

2 input pointing device capable of sensing motion in at least three dimensions and a pressure sensor, 

3 the first and second computer input pointing devices being connected to the USB hub, wherein each 

4 of the first and second computer input pointing devices controls a respectiw animated character. 

1 16. The computer system of claim 14, wherein the computer program code includes a range 

2 of motion tree for the animated character, the range of motion tree restricts a range of possible 

3 motion of the animated character about a current position of the animated character. 

1 17. The computer system of claim 16, wherein the computer program code includes a 
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1 18. The computer system of claim 16, wherein the computer program code includes a module 

2 for using output signals from the computer input pointing device to emulate output signals from a 

3 mouse. 

1 19. The pointing device of claim 4, wherein each of the three piezoelectric strips has at least two 

2 sections, including a weighted section and an unweighted section. 

1 20. The pointing device of claim 5, wherein the cut coils on each face of the motion detector are 

2 attached to a single weight at a center of the motion detector. 

1 21. The pointing device of claim 1, further comprising a multiplexer that receives output signals 

2 from the motion detector and the pressure sensor, for outputting a multiplexed signal for 

3 transmission to a computer, wherein the motion detector, pressure sensor and multiplexer are 

4 contained within a squeezable housing. 

1 22. The pointing device of claim 7, wherein the squeezable housing has at least one tactile feature 

2 to provide a tactile cue for orienting the pointing device. 

1 23. The pointing device of claim 7, wherein the squeezable housing has at least one visible 

2 feature to provide a visual cue for orienting the pointing device. 

1 24. A method for operating a computer having an output device, comprising the steps of: 

2 receiving a plurality of signals from a computer input pointing device, the signals 

3 representing quantifiable pressure and motion in at least three dimensions; and 

4 outputting a video, audio or tactile output signal to the output device, the output signal 

5 having at least two characteristics that are capable of being varied separately from each other. 

1 25. The method of claim 24, wherein: 

2 the output device is a display; 

3 the step of outputting includes displaying an animated character on the display, and 

4 moving at least two portions the animated character separately from each other. 
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1 26. A method for operating a computer input pointing device, comprising the steps of: 

2 moving the pointing device; 

3 transmitting at least three motion signals from the pointing device to a computer, the motion 

4 signals including signals representing translations in one or more dimension, or rotations in one or 

5 more dimensions, or a combination of translations and rotations; 

6 squeezing at least one portion of the pointing device; and 

7 transmitting quantifiable pressure signals from the pointing device to the computer. 

1 27. The method of claim 26, further comprising the steps of: 

2 speaking into a microphone contained within the pointing device; and 

3 transmitting an audio signal from the microphone to the computer. 
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